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Abstract 


The  US  Air  Force  Small  Business  Innovation  Research  (SBIR)  program  is  a  vital  element  of  the  Air  Force 
Research  Faboratory  (AFRF)  portfolio.  The  SBIR  program  funds  early-stage  R&D  projects  at  small  technology 
companies  that  support  a  US  Department  of  Defense  (DoD)  need  and  have  the  potential  for  commercialization  in 
private  sector  and/or  military  markets.  We  report  on  research  measuring  the  application  of  Systems  Engineering 
(SE)  and  the  degree  of  Systems  Engineering  applied  in  SBIR  projects  through  analysis  of  subject-matter  expert 
(SME)  interviews.  SMEs  were  sampled  front  AFRL  as  well  as  other  USAF  organizations.  The  research 
methodology  assessed  the  current  application  of  SE  in  SBIR  as  well  the  SE  expectations  government  offices  have 
for  SBIR  projects.  The  research  specifically  examined  current  US  DoD  and  US  Air  Force  Systems  Engineering 
Policy  and  found  that  it  does  not  adequately  address  SBIR  projects.  SE  processes  are  not  well  documented  within 
the  community.  We  then  identified  applicable  SE  tasks  found  in  Air  Force  SE  policy  and  suggest  a  tailored  Systems 
Engineering  approach  for  SBIR  projects.  We  present  a  SBIR  SE  checklist  recommended  for  use  by  SBIR  program 
managers  tailored  according  to  the  phase  and  scope  of  any  SBIR  project. 
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1.  Introduction 

The  US  DoD’s  SBIR  program  is  a  large  part  of  the  multibillion  dollar  federal  SBIR  program  administered 
by  twelve  federal  agencies  across  the  country  [www.sbir.gov/about/about-sbir].  The  SBIR  program  funds  early- 
stage  R&D  projects  at  small  technology  companies  that  support  a  US  government  agency  need  and  have  the 
potential  for  commercialization  in  the  private  sector  and/or  military  markets.  SBIR  projects  are  managed  by  many 
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different  organizations  throughout  the  DoD.  Within  the  Air  Force,  SBIR  projects  are  managed  by  AFRL 
Technology  Directorates,  Test  Centers  and  Air  Logistics  Centers.  The  Air  Force  Small  Business  Innovation 
Research  (SBIR)  program  is  vital  element  of  the  Air  Force  Research  Laboratory  (AFRL)  contracts  portfolio 
operated  under  the  guidance  of  the  Air  Force  SBIR/Small  Business  Technology  Transfer  (STTR)  Program  Manager 
within  AFRL  at  Wright  Patterson  Air  Force  Base.  The  DoD  and  Air  Force  provide  top  level  SE  guidance  and  policy 
for  the  acquisition  community.  However  the  current  guidance  and  policy  has  not  yet  been  tailored  specifically  for 
the  SBIR  community.  This  research  explores  this  gap  and  potential  solutions  in  an  applied  setting  at  the  AFRL.  The 
focus  of  this  paper  is  to  present  our  assessment  of  SE  applicability  to  SBIR  programs  and  not  address  broad  SE  and 
acquisition  issues  that  are  beyond  the  scope  of  S&T  programs. 

The  DoD  has  well  defined  system  engineering  processes  documented  in  the  department’s  5000  series  of 
instructions  on  acquisition.  The  guidance  cited  in  this  paper  applies  to  both  the  AFRL  SBIR  project  manager,  since 
his  responsibilities  include  systems  engineering,  and  the  SBIR  awardee  because  both  are  part  of  the  S&T  acquisition 
community.  However  a  number  of  challenges  exist  with  applying  SE  to  SBIR  projects  since  they  are  unique 
compared  to  typical  acquisition  programs.  They  are  managed  by  many  different  small  businesses  that  may  or  may 
not  have  an  organic  SE  capability.  Additionally  they  vary  significantly  in  scope,  are  small  in  size,  short  in  project 
length,  and  are  early  research  projects.  They  are  categorized  as  Basic  Research  or  Applied  Research  projects. 

Topics  are  generated  across  the  Air  Force  by  Program  Executive  Officers,  Technology  Directorates,  Air  Logistics 
Centers  and  Test  Centers.  A  SBIR  project  is  developed  in  three  phases.  Phase  1  is  a  technical  feasibility  study  that 
allocates  up  to  $150k  and  9-12  months  to  completion.  Phase  II  is  concept  development  and  allocates  up  to  $1M  and 
24  months  to  completion.  There  are  also  Critical  Manufacturing  SBIR  projects  that  are  allocated  up  to  $5M  for 
Phase  II.  Phase  III  is  the  commercialization  stage  [www.sbir.gov].  As  SBIR  projects  vary  considerably  in  scope,  are 
managed  by  many  different  organizations  within  the  government,  and  work  is  accomplished  by  varying  small 
businesses,  a  problem  of  consistent  SE  processes  being  applied  across  all  projects  is  presented. 

1.2  Research  Focus 

We  seek  to  identify  how  current  SE  practices  apply  to  SBIR  projects.  This  includes  identifying  how  and 
what  current  DoD  and  Air  Force  SE  policies  apply  to  SBIR  projects  during  Phase  I  and  II  and  how  to  best  tailor  the 
guidance  to  develop  a  solid  SE  approach  for  the  technical  management  of  the  projects.  Thus  the  subject  of  this 
research  focuses  on  implementation  of  early  SE  processes  for  Phase  I  and  II  SBIR  projects.  We  suggest  without  a 
solid  SE  approach  SBIR  projects  are  at  risk  to  fail,  while  good  SE  processes  will  help  ensure  projects  are  better 
prepared  for  proceeding  to  their  next  phase  of  development  while  adequately  managing  technical  risk. 

2.0  Background  and  Literature  Review 

The  US  Department  of  Defense  publishes  the  Defense  Acquisition  Guide  (DAG)  and  the  DoD  5000  series 
of  instruction  to  identify  SE  processes.  The  Air  Force  has  also  published  a  series  of  instructions.  Air  Force 
Instruction  (AFI)  63-1201  “Life  Cycle  SE”  is  for  the  acquisition  community.  AFI  63-1201  defines  SE 
responsibilities  for  program  managers  and  engineers.  Additionally,  Air  Force  Material  Command,  the  organization 
in  the  Air  Force  most  responsible  for  system  development,  created  the  Air  Force  SE  Assessment  Model  (AF  SEAM) 
for  assessment  of  Air  Force  programs.  SEAM  also  contains  goals  and  specific  as  well  as  general  practices  that  can 
be  used  in  assessment.  Differences  between  the  DAG,  AFI  63-1201  and  AF  SEAM  can  be  confusing  since  they  vary 
in  terminology.  The  tables  below  identify  the  SE  processes  identified  in  each  document.  Oneillustrates  how  the 
DAG  has  sixteen  distinct  processes,  AF  63-1201  has  twelve  processes  and  AF  SEAM  has  ten,  and  the  other  shows 
the  breakout  of  SEAM  into  goals  and  practices. 
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Table  1:  SE  Processes  (AF  SEAM,  Sept  2010) 


AT  SEAM  Defeat,  Acquisition  Guide  ATI  «3  1201 


Requirements 

Keqts  Analysis,  Krqls  Mgmt. 
Stakeholder  Reqts  Definition 

Keql  Des  &  Mgmt.  & 
Architecture 

Design 

Architectural  Design. 
Integration  &  Interface 
Mgmt 

Design  &  Interface  Mgmt 

Verification  &  Validation 

Verification  &  Validation 

Test  &  Evaluation, 
Verification  A  Validation 

Mnuufacturiug 

Implementation 

Desigu 

t  ransition.  Fielding.  & 
Sustainment 

Transition 

Design 

Project  Planning 

Technical  Planning 

Planning 

Configuration 

Management 

CM,  Data  Mgmt.  Technical 
Data  Mgmt 

C  ouflguration  Mgmt.  Data 
Mgint 

Risk  Management 

Risk  Mgmt 

Integrated  Risk  Management 

Technical  Mgmt  & 
Control  (PMC) 

Technical  Assessment 

Technical  Reviews  & 
Measurements 

Derision  Analysis 

Decision  Analysis 

Decision  Analysis 

Table  2:  AF  SEAM  Tasks  (AF  SEAM,  Sept  2010) 


AF  SEAM  Tasks 

Process  Area 

Goals 

Specific 

Practices 

Generic 

Practices 

Total 

Practices 

Configuration  Mgmt 

3 

8 

7 

16 

Decision  Analysis 

1 

5 

7 

12 

Design 

3 

14 

7 

21 

Manufacturing 

4 

12 

7 

ie 

Project  Planning 

3 

15 

7 

22 

Requirements 

4 

13 

7 

21 

Risk  Mgmt 

3 

7 

7 

14 

Trans.  Fielding.  &  Sus 

4 

15 

7 

22 

Tech  Mgmt  &  Control 

4 

15 

7 

18 

Verification  &  Validation 

5 

16 

7 

23 

Total 

34 

120 

70 

190 

AFI  63-1201  is  currently  under  revision  to  better  align  with  AF  SEAM  and  the  DAG.  The  revised  version 
of  AFI  63-1201  should  reduce  misalignment  by  eliminating  the  current  differences  with  AF  SEAM. 

2.1  Air  Force  Systems  Engineering  Assessment  Model  (SEAM) 

The  primary  purpose  of  AF  SEAM  is  to  promote  the  application  and  use  of  standard  SE  processes  across 
the  AF  and  to  improve  the  performance  of  these  processes  through  Continuous  Process  Improvement  [AF  SEAM, 
September  2010].  AF  SEAM  is  not  yet  mandated  across  the  Air  Force  however  it  is  used  as  a  reporting  tool  in  some 
AF  communities.  AF  SEAM  identifies  “ten  AF  standard  SE  process  areas”  and  lists  associated  goals  for  each. 
Specific  practices  and  generic  practices  are  identified  for  each  area.  Table  2  identifies  190  total  practices  in  AF 
SEAM.  This  suggests  a  significant  SE  effort  is  required  for  any  program  to  implement  AF  SEAM. 

2.2  AFRL  Policy 

The  Air  Force  Research  Labs  have  two  main  documents  providing  guidance  for  SE.  The  first  is  AFRL 
Instruction  61-104  which  addresses  SE  in  the  S&T  environment.  It  is  in  alignment  with  DoD  and  Air  Force  policy 
but  tailors  it  to  the  S&T  community.  It  also  identifies  Eight  Systems  Engineering  Key  Questions  to  guide  and  assess 
the  SE  health  of  a  project.  These  are  mapped  back  to  the  DAG  SE  process  areas.  The  second  document  is  AFRL  61- 
204,  a  Manual  for  Scientist  and  Engineers.  An  attachment  identifies  “What  the  Program  Manager  should  know 
about  his  or  her  program.”  It  says  “Use  of  the  key  questions  during  reviews  of  basic  research  programs  is  optional.” 
Also  in  an  attachment,  AFRL  translates  the  16  SE  DAG  processes  for  the  AFRL  community  as  shown  in  Figure  1.  It 
defines  each  process  for  AFRL  and  explains  the  importance.  The  AFRI  Instruction  specifically  declares  it  is  tailored 
for  a  typical  AFRL  program  and  not  for  SBIR  programs. 


Figure  1.0  The  S&T  SE  Process. 
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Figure  1:  AFRL  S&T  SE  Process  (Draft  AFRL  SE  Guidebook,  2012) 
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Several  theses  and  past  studies  have  been  accomplished  analyzing  SE  efforts  within  AFRL.  A  past  research 
project  “A  Tailored  SE  Framework  for  S  &  T  Projects”  captures  the  SE  tasks  and  rigor  applicable  for  typical  AFRL 
projects  (Pitzer,  Behm  and  White,  2009).  They  developed  a  tool,  the  “Systems  Engineering  Tailoring  tool  for 
Science  &  Technology  Projects,”  that  defined  projects  by  6  parameters:  RDT&E  Category,  Project  Budget,  Core 
Process,  Technology  Readiness  Level,  Integration  Level,  and  Requirements  Maturity.  The  tool  suggests  what  SE 
best  practices  (mapped  from  the  16  DAG  processes)  would  apply  to  that  project/program.  This  tool  is  notably 
similar  to  AF  SEAM  however  it  tailors  the  tasks  for  a  project  based  on  the  stated  parameters.  Additionally  four 
notable  AFRL  studies  are  significant  for  this  research: 

1 .  High  Energy  Laser  On  a  Large  Tactical  Platform  (HELLTP) 

2.  Deployed  Base  Energy  Alternatives  Report 

3.  Company  Grade  Officer  Initiative  Program  (CGOIP) 

4.  AFRL  Transformational  Activities  in  Systems  Engineering  (TASE)  Assessment  Phase  Final  Report 

Findings  from  2006 

The  first  two  focus  on  the  successful  tailoring  and  streamlining  of  SE  efforts  on  two  larger  AFRL  projects. 
The  third  focused  on  tailoring  and  streamlining  SE  efforts  of  smaller  projects  in  CGOIP.  This  study  was  notable 
since  the  projects  were  being  managed  by  CGOs  with  limited  SE  backgrounds  -  not  unlike  SBIR  project  managers. 
And  like  the  first  two  studies  listed,  CGOIP  was  deemed  successful  in  implementing  good  SE  processes  into  their 
projects  using  a  streamlined  SE  approach.  The  final  study  focused  on  making  AFRL  research  programs  more 
effective  and  efficient  and  by  improving  the  transition  of  technology  to  the  warfighting  community  through  the  use 
of  good  SE  processes. 

The  studies  identified  common  key  SE  attributes:  forming  multi-disciplinary  teams  involving  all  relevant 
stakeholders;  holding  team  reviews  to  monitor  project  progress;  and  tailoring  their  SE  approach  with  a  streamlined 
S&T  process  consistent  with  Air  Force  policy. 

2.4  Summary  of  Literature 

Review  of  SE  policy  within  the  DoD,  Air  Force  and  AFRL  has  identified  defined  SE  guidance  and 
instruction.  AFRL  has  done  a  good  job  tailoring  their  policy  to  be  in  alignment  with  higher  level  guidance  and 
continues  to  be  proactive  in  implementing  good  SE  processes  within  AFRL.  However,  the  literature  does  not 
specifically  address  the  application  of  Systems  Engineering  to  SBIR.  This  constitutes  a  knowledge  gap  in  the 
understanding  of  systems  engineering. 

Subsequent  discussions  with  AFRL  officials  recognized  that  AF  SEAM  is  a  rigorous  tool  for  implementing 
SE  but  is  truly  not  tailored  for  the  Science  and  Technology  (S&T)  community.  In  its  current  configuration,  many  of 
the  190  SE  tasks  may  not  apply  given  the  attributes  of  the  AFRL  project.  Further,  implementing  this  model  “as  is” 
does  not  make  sense  for  the  SBIR  community  either  due  to  the  more  unique,  further  limited  resources  and 
questionable  value  these  tasks  will  add  to  a  project.  A  more  tailored  approach  is  required. 

Preliminary  research  within  the  SBIR  community  identified  varying  levels  of  SE  knowledge  and 
implementation  amongst  project  managers.  SE  guidance  needs  to  be  better  defined  and  tailored  for  the  SBIR 
community  to  ensure  SE  deliverables  are  sufficient  for  transitioning  the  project  to  the  next  phase  of  research. 

The  SETT  tool  provides  a  good  initial  baseline  for  an  AFRL  sponsored  project,  however,  SBIR  projects  do 
not  fit  the  traditional  project  mold.  Implementing  a  process  or  tool  that  is  not  tailored  to  the  appropriate  level  risks 
creation  of  non-value  added  work  and  can  drain  valuable  resources  from  a  project.  Nevertheless,  both  the  SETT 
Tool  and  AF  SEAM  will  be  good  baselines  to  consider  when  identifying  what  SE  tasks  may  apply  to  SBIR  projects. 

Finally,  the  various  studies  underscored  the  need  to  develop  a  tailored  SE  approach  for  the  SBIR 
community.  They  identify  many  weaknesses  in  the  S&T  community  and  suggest  some  policy  changes  for  good 
implementation  of  SE  processes. 

3.0  Methodology 

A  reference  framework  using  the  material  identified  in  the  review  of  literature,  SBIR  documentation  and 
applicable  SE  policy  was  created.  In  most  cases  there  was  little  if  any  SE  documentation  available.  Therefore,  using 
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a  mixture  of  qualitative  and  quantitative  methods  this  research  seeks  to  develop  a  tailored  system  engineering 
approach  for  SBIR  projects  as  an  exploratory  work.  Inductive  theory  or  grounded  theory  development  is  the  key 
methodological  component.  Interviews  were  used  to  gather  additional  information.  This  approach  also  helped 
identify  current  data  requirements  and  degree  of  SE  rigor  currently  associated  with  SBIR  projects.  Then  interview 
and  literature  data  was  analyzed  through  structured  content  analysis  using  data  triangulation  methods  to  identify  the 
relevant  SE  practices  for  SBIR.  Triangulation  methods  in  data  analysis  improves  the  validity  and  reliability  of  this 
type  of  research  [Glafshani,  2003]. 

Specifically,  the  interview  instrument  to  collect  data  from  SBIR  project  managers  is  semi-structured  so  that 
open-ended  responses  are  encouraged  and  snowball  sampling  could  occur.  Interviewees  were  identified  from 
different  AFRL  and  external  AF  organizations  based  upon  their  position  in  the  organization  and  familiarity  with  the 
SBIR  process.  This  group  included  several  SBIR  program  managers.  Once  all  interviews  were  complete  and  data 
had  been  collected  the  results  were  analyzed  to  help  suggest  the  appropriate  SE  tasks  for  SBIR  projects.  Figure  2 
represents  the  flow  of  the  methodology  used  in  this  research. 


Figure  2:  Methodology 


3.1  Research  Objectives  and  Hypothesis 

One  objective  of  this  research  project  is  to  define  the  SE  rigor  that  should  be  best  applied  to  SBIR  projects. 
To  define  the  rigor  and  design  a  tailored  approach  requires  identifying  what  degree  of  SE  is  applicable  in  the  SBIR 
environment  and  how  those  SE  processes  vary  amongst  projects  with  respect  to  project  maturity,  size  and  other 
factors. 

A  working  hypothesis  is  that  different  organizations  implement  varying  levels  of  SE  into  their  SBIR 
projects  based  upon  some  of  their  defined  needs.  Furthermore,  organizational  SE  policies  and  the  SE  knowledge 
base  vary  amongst  SBIR  project  managers.  Therefore,  these  projects  carry  some  risk  to  meet  DoD  and  AF  standards 
required  by  policy  and  instruction  in  the  implementation  of  SE  and  SE  processes. 

3.2  The  Interview  Instrument  and  Sample  Selection 

The  interview  was  designed  to  investigate  current  AFRL  policy.  One  question  for  each  SE  process  outlined 
in  the  DAG  was  created  corresponding  with  the  specific  tasks  called  out  in  AFRL  policy.  Additional  questions  were 
asked  regarding  the  effectiveness  of  the  Eight  SE  Key  Questions  outlined  in  AFRL  policy. 


Table  3:  Total  Interview  Participants 


Technology  Directorates 

Test 

Centers 

Air  Logistics 
Centers 

Other 

Materials  & 
Manufacturing 

Propulsion 

Space 

Vehicles 

Human 

Effectiveness 

Munitions 

Arnold 

Robins 

ASC 

4 

7 

3 

2 

2 

2 

3 

1 

Table  3  indicates  the  total  numbers  of  personnel  interviewed  and  the  different  organizations  they  represent. 
The  ideal  participants  for  the  interview  were  identified  as  SBIR  Program  Managers  (PM),  engineers  (EN)  and  AFRL 
organizational  chief  engineers  (CE)  because  they  are  the  most  familiar  with  the  daily  technical  management  of  a 
SBIR  project.  The  goal  was  to  interview  several  PMs,  ENs,  and  CEs  from  each  organization.  SBIR  project 
managers  were  a  natural  choice  as  they  perform  day-to-day  management  of  the  SBIR  project.  These  project 
managers  are  engineers  or  program  managers  with  varying  levels  of  experience.  Chief  Engineers  of  the  directorates 
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oversee  the  management  of  these  projects  within  AFRL.  Participants  for  the  interview  were  identified  through 
purposeful  and  snowball  sampling. 


4.0  Analysis  and  Results 

Interview  results  are  analyzed  in  terms  of  the  tasks  identified  from  the  DAG.  Then  the  findings  surrounding 
the  8  SE  Key  Questions  are  discussed  and  finally,  the  results  relative  to  AF  SEAM  are  given.  Figure  3  below 
illustrates  the  percentage  of  positive  responses  for  the  tasks  (ranging  from  two  to  five  for  each)  identified  from 
AFRL  Policy  captured  in  the  interview  for  each  SE  process  area.  For  areas  with  more  than  50%  task  use,  green  is 
shown;  yellow  is  used  showing  30%  to  50%  and  red  for  areas  with  less  than  30%  task  use  in  each  SE  process  area. 
Roughly  speaking,  participants  identified  that  only  a  total  of  55%  of  the  SE  tasks  listed  in  the  survey  were  applicable 
to  SBIR  projects. 
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Figure  3:  Results  from  Test  Centers,  Technology  Directorates,  and  Air  Logistic  Centers 
(Horizontal  axis  is  the  same  on  all  charts  -  abbreviated  on  last  two  for  space) 

Test  Center  interviewees  complete  42%  of  SE  Tasks.  No  interviewees  identified  the  SE  tasks  of 
Architectural  Design,  Decision  Analysis  and  Interface  Management  as  applicable.  Since  the  interview  instrument 
was  developed  using  AFRL  policy,  we  believe  some  of  the  tasks  were  couched  in  AFRL  language  and  participants 
did  not  necessarily  identify  with  them. 

Technology  Directorate  interviewees  complete  62%  of  SE  Tasks.  This  is  notable  for  the  Technology 
Directorates  because  so  many  of  the  SE  tasks  identified  in  AFRL  policy  are  not  accomplished  during  Phase  I  &  II  of 
SBIR  projects.  Additionally  several  areas  of  weakness  are  seen  in  areas  where  few  tasks  are  completed  in  a 
particular  DAG  categorized  SE  area.  This  may  illustrate  how  SBIR  projects  are  unique  when  compared  to  typical  S 
&  T  projects  within  AFRL. 

Air  Logistics  Center  interviewees  complete  35%  of  SE  Tasks.  As  before,  since  the  interview  instrument 
was  developed  using  AFRL  policy,  we  believe  some  of  the  tasks  were  couched  in  AFRL  language  and  participants 
did  not  necessarily  identify  with  them.  Preliminary  analysis  of  these  results  also  suggests  that  Air  Logistics  Centers 
and  Test  Centers  have  a  different  focus  or  purpose  in  their  SBIR  projects. 

4.1  Value  of  the  AFRL  Eight  Systems  Engineering  Key  Questions 

Only  those  interviewees  from  AFRL  organizations  were  asked  about  the  value  and  usefulness  of  the  Eight 
SE  Key  Questions  identified  in  AFRLI  61-104.  Table  4  shows  the  qualitative  value  interviewees  associated  with  the 
questions. 


Table  4:  Value  of  the  8  AFRL  SE  questions  from  interviewee  responses 


Didn't 
know  they 
existed 

not 

useful 

a  little 
usefiil 

Some¬ 

what 

useful 

moderately 

useful 

useful 

very 

useful 

extremely 

useful 

Total 

Participants 

8  SE  Key 
Questions 

4 

1 

1 

2 

0 

2 

3 

2 

15 

Almost  seventy  percent  of  participants  who  knew  of  the  SE  questions  thought  they  were  useful  to  some 
degree.  No  definitive  conclusions  about  the  value  of  these  questions  are  drawn  due  to  the  small  sample  size,  but 
there  is  concern  that  these  questions  were  completely  unknown  to  several  interviewees.  Table  5  shows  how  each 
participant  subjectively  identified  the  most  and  least  useful  of  the  8  SE  questions.  Some  interviewees  gave  multiple 
answers  on  the  most  useful  questions  while  others  did  not  identify  any  “Least  Useful”  questions. 
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Table  5:  AFRL  SE  Key  Questions  Mapped  to  Interview  Results 


Most  Useful 

Least  Useful 

1 .  Who  is  your  customer? 

5 

2 

2.  What  are  the  customer’s  requirements? 

5 

3 .  How  will  you  demonstrate  you  have  met  the  requirements? 

5 

1 

4.  What  are  the  technology  options? 

1 

2 

5 .  Which  is  the  best  approach? 

1 

6.  What  are  the  risks  to  developing  the  selected  technology? 

2 

1 

7.  How  will  you  structure  your  program  to  meet  requirements  and  mitigate  risk? 

2 

8.  What  is  your  business-based  transition  plan  that  meets  customer  approval? 

4 

Participants  felt  that  the  first  three  questions  were  the  most  important  and  the  least  important  question  was 
question  8.  This  is  likely  because  not  all  SBIR  projects  transition  into  larger  projects.  These  responses  warrant 
further  investigation  as  conclusions  cannot  definitively  be  draw  based  upon  the  interview  design. 

4.2  AF  SEAM  Comparison 

Comparing  the  interview  results  with  AF  SEAM  the  applicable  SE  tasks  for  the  SBIR  community  were 
identified.  However,  as  discussed  earlier,  AF  SEAM  and  AFRL  policy  do  not  directly  align  since  AFRL  policy 
maps  back  to  the  DAG.  Only  about  50%  of  the  tasks  from  AF  SEAM  were  found  to  be  applicable  for  the  SBIR 
community.  Many  of  the  tasks  identified  in  AF  SEAM  are  not  applicable  until  the  later  phases  of  a  program. 

Further,  AF  SEAM  requires  a  large  manpower  effort  to  complete,  due  to  the  sheer  number  of  its  SE  tasks  (190). 
Therefore  it  would  not  be  useful  to  implement  AF  SEAM  within  the  SBIR  community,  without  tailoring. 

5.0  Conclusions  and  Recommendations 

Based  upon  the  analysis  and  results,  a  SBIR  SE  Checklist  (see  Table  6)  was  developed  as  a  guide  for 
project  managers  and  engineers  to  ensure  SE  is  adequately  addressed  in  SBIR  projects.  This  checklist  also  aligns 
with  the  10  AF  SEAM  SE  processes  areas.  Those  using  this  checklist  should  begin  to  accomplish  some  of  these 
tasks  in  Phase  1  with  the  emphasis  of  demonstrating  the  project  is  feasible,  identifying  stakeholders  and  defining 
requirements.  For  instance,  it  is  critically  important  that  AFRL  project  managers  clarify  AF  requirements  or  basic 
research  objectives.  Small  businesses  do  not  usually  possess  organic  resources  to  determine  DoD  requirements. 
AFRL  project  managers,  though,  are  in  a  position  to  get  access  to  capability  gaps  or  engage  AF  Programs  of  Record 
or  contact  other  key  contractors  for  more  information  on  AF  requirements.  By  the  end  of  Phase  2  all  of  the  listed 
tasks  should  have  been  tailored  to  the  specifics  of  each  SBIR  project  and  accomplished.  Projects  that  enter  Phase  2.5 
should  emphasize  further  defining  and  documenting  information  and  demonstrating  the  technology  with  the  hopes  to 
aid  in  the  future  transition  of  the  project  to  Phase  III. 

The  following  symbols  are  used  to  interpret  the  checklist: 

•  Represents  general  AF  SE  process  tasks  tailored  for  SBIR 
Represents  specific  SE  tasks  captured  in  analysis 


Table  6:  SBIR  SE  Checklist 


Requirements 

•  Determine  requirements  to  include  stakeholder  needs,  expectations,  constraints,  and  interface  requirements. 

Translate  all  stakeholder  needs  to  technical  requirements. 

Requirements  made  quantifiable,  have  unique  definitions,  and  specified  thresholds  and  objectives. 

Work  with  stakeholders  to  refine  requirements. 

Perfonnance  parameters  and  constraints  allocated  and  derived  technical  requirements  defined. 

Maintain  the  traceability  of  all  requirements  from  needs. 

Document  changes  and  record  rationale  of  changes. 

Project  Planning 

•  Identify  project  milestones  to  include  cost,  schedule  and  technical  milestones. 

Define  the  scope  of  the  tech  effort  required  to  achieve  program  technical  goals. 

Define  exit  criteria  and  products/deliverables  which  can  be  tracked  with  progress  measured. 
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Risk  Management 

•  Develop  a  risk  management  plan  and  identify,  analyze,  identify  handling  options,  mitigate  and  track  risk. 

Decision  Analysis 

•  Establish  selection  criteria,  identify  &  evaluate  alternatives  and  select  solution. 

Criteria  selected  for  decision  &  methods  to  be  used  in  conducting  the  analysis. 

Identify  analysis  methods  and  conduct  analysis  of  alternatives. 

Design 

•  Establish  the  design  and  integration  baseline. 

Incorporate  the  lower-level  system  elements  into  a  higher-level  system  element  in  the  physical  architecture. 
Indentify  constraints  &  interfaces  that  will  result  in  derived  technical  requirements. 

Technical 

Management  & 

Control 

•  Establish  and  maintain  the  project  environment,  integrated  product  teams  (IPT),  measurements  approach 

and  monitor  technical  reviews,  work  products,  project  data,  corrective  actions  and  technical  milestones. 
Measure  technical  progress,  technology  maturity  and  the  effectiveness  of  plans  and  requirements. 
Demonstrate  and  confirm  completion  of  required  accomplishments  and  project  exit  criteria. 

Configuration 

Management 

•  Establish  the  technical  baseline,  track  and  document  changes. 

Maintain  record  of  all  configurations  to  include  hardware,  software  and  test  set  up  and  document  changes. 

Define  internal  and  external  interfaces. 

Project  data  managed  through  the  Defense  Technical  Information  Center. 

Verification  & 

Validation 

•  Establish  and  maintain  the  overall  verification  strategy  and  plan  to  include  verification  and  validation 

criteria  and  an  integrated  testing  approach  when  applicable.  Verify  and  Validate  that  the  project  has  meets 
the  required  parameters. 

Confirm  project  meets  design  specifications. 

Test  the  system  elements  against  their  defined  requirements. 

Test  the  performance  of  the  technology  against  the  original  program  goals. 

Transition,  Fielding, 

&  Sustainment 

•  Indentify  future  transition,  fielding,  &  sustainment  requirements  as  needed  to  proceed  to  the  next  phase  of 

the  project. 

Needs  of  follow-on  phases  considered  early  in  the  program  and  included  in  all  of  the  technical 
management  processes. 

Yield  the  fundamental  capability  of  the  program. 

Manufacturing 

•  Identify  and  maintain  documentation  relevant  to  the  future  production  of  the  project. 

Develop  supporting  documentation  for  the  system. 

Current  policy  does  not  fully  define  SE  for  the  SBIR  community  and  SE  is  being  implemented  at  various 
levels  amongst  the  different  organizations  that  manage  SBIR  projects.  Leadership  and  project  managers  must  ensure 
adequate  levels  of  SE  are  being  incorporated  into  their  projects  to  improve  their  chance  of  success,  limit  cost  and 
schedule  overruns  and  meet  performance  goals.  Good  implementation  of  SE  processes  can  be  measured  by  the 
smooth  successful  transition  of  these  projects  to  the  next  phase  of  development.  Completion  of  all  SE  tasks  in  this 
checklist  can  be  used  as  a  way  to  measure  implementation  of  SE  processes  for  each  SBIR  project.  Failure  to  follow 
established  SE  processes  in  any  one  area  can  have  significant  negative  consequences  to  the  project. 

Specifically,  the  research  identified: 

•  The  SBIR  community  does  not  believe  all  SE  tasks  in  AFRL  policy  apply  to  their  projects 

•  The  SBIR  community  does  not  consistently  implement  SE  tasks. 

•  There  is  a  wide  spectrum  of  interpretation  that  is  partially  due  to  the  sponsoring  organization  type  and  the 
phase  of  the  SBIR  project.  Those  results  identified  weak  areas  within  the  current  policy. 

•  DoD  and  Air  Force  SE  processes  must  be  tailored  for  the  scope  of  the  SBIR  project. 

•  Applicable  SBIR  SE  tasks  identified  by  this  research  are  shown  in  Table  6  above. 

5.1  Significance  of  Research 

This  research  revealed  a  SE  policy  gap  in  the  SBIR  community  and  defined  applicable  SE  tasks.  This 
represents  a  huge  risk  as  millions  of  dollars  are  spent  within  the  DoD  each  year  on  SBIR  projects.  Failure  to 
implement  good  SE  principles  can  and  will  lead  to  cost  overruns,  schedule  slips  and  perfonnance  short  falls. 
Findings  from  this  research  should  be  used  to  tailor  a  SE  approach  for  SBIR  projects  to  ensure  SE  practices  are 
being  implemented  in  a  best  practice  manner. 
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5.2  Assumpl  ions/Li  mi  tat  ions/I  triplications 

Since  organizations  managing  SBIR  projects  are  geographically  separated  it  was  not  feasible  to  gather  data 
from  enough  organizations  to  have  representative  data  for  all  SBIR  projects.  Therefore,  this  study  is  based  on 
representative  sampling.  Using  a  larger  sample  size  to  verify  the  conclusions  drawn  here  is  recommended. 

As  stated  above,  the  SBIR  has  a  better  chance  for  transition  if  the  SBIR  project  manager  and  the  SBIR 
awardee  perform  Table  6  tasks.  SBIR  Phase  I  should  focus  on  requirements,  project  planning,  and  risk 
management.  SBIR  Phase  II  and  Phase  II  extensions  should  apply  all  Table  6  tasks.  The  tasks  are  not  difficult  and 
would  provide  useful  information  to  PMs  and  awardees  early  in  the  SBIR  project  instead  of  dealing  with  larger 
issues  downstream.  A  follow-on  paper  could  explore  more  closely  the  details  associated  with  optimal  SE  work 
breakdowns  for  AFRL  project  engineers  and  R&D  contractors. 

Though  this  project  focuses  specifically  on  SBIR  projects,  findings  will  likely  be  applicable  to  similar  S&T 
projects.  Projects  in  early  development  will  have  many  similar  attributes  to  the  SBIR  projects  analyzed  in  this 
research.  This  work  could  be  used  to  guide  a  tailored  SE  approach  for  similar  projects/programs.  A  follow-on  paper 
could  explore  the  applicability  of  Table  6  to  other  R&D  projects/programs  including  in-house  programs. 

5.3  Recommendations  for  Action 

First,  organizational  policy  needs  to  be  tailored  for  SBIR.  The  SBIR  community  should  use  the  identified 
SBIR  SE  applicable  tasks  from  this  study  to  develop  adequate  policy  and  SE  tasks  for  their  SBIR  projects. 

Second,  the  SBIR  community  should  incorporate  a  tailored  SE  approach  for  their  projects.  The  literature 
also  identifies  the  benefits  of  using  such  an  approach. 

Finally,  the  SBIR  community  should  ensure  the  project  managers  receive  adequate  SE  education  to  enable 
them  to  tailor  SE  to  their  projects.  As  the  scope  of  SBIR  projects  can  vary  greatly  it  can  be  challenging  for  project 
managers  to  understand  how  all  areas  of  SE  apply  to  their  projects. 
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